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ABSTRACT 


Hybrid  knowledge  bases  (HKBs),  proposed  by  Nerode  and  Subrahmanian,  provide  a  uniform 
theoretical  framework  for  dealing  with  the  mixed  data  tynes  and  multiple  reasoning  modes  required 
for  solving  logical  deployment  problems.  Algorithms  based  on  mixed  integer  linear  programming 
techniques  have  been  developed  for  the  syntactic  subset  of  HKBs  corresponding  to  function-free 
Prolog-like  logic  programs.  In  this  study,  we  examine  the  ability  of  neitf^^tworks  to  solve  a  more 
comprehensive  set  of  problems  expressed  within  the  hybrid  knowledge  base  framework. 


The  objective  of  this  research  is  to  design  and  implement  a  nonlinear  optimization  procedure 
for  solving  extended  togjcprograms  with  neural  networks.  We  focus  upon  two  types  of  extensions 
which  are  typically  required  in  the  formulation  of  The  first  type 

of  extension,  which  we  shall  refer  to  as  a  Type  I  extension,  consists  of  embedding  numerical  and 
tn^onstmnts  into  logic  programs.  The  second  type  of  extension,  which  we  shall  call  a  Type 
extension,  consists  of  incorporating  optimiz|tionproblems  into  logic  clauses. 


1.  INTRODUCTION 

A  potentially  useful  way  of  organizing  the  massive  volume  of  terrain  information  available 
from  multiple  sensors  with  human  expertise  and  decision-making  criteria  is  through  facts  and  rules 
that  can  be  accessed  by  logic  programs.  In  these  systems  numerical,  geographical,  and  geometric 
information  must  be  combined  uniformly  with  logical  information  as  in  human  inference  to  solve 
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logical  deployment  problems.  The  rules  in  the  system  must  be  able  to  express  various  types  of 
relationships,  including  constraint  relationships  and  relationships  framed  as  optimisation  problems. 
Hybrid  knowledge  bases1,  proposed  by  Nerode  and  Subrahmanian,  provide  a  general  mechanism  for 
uniformly  dealing  with  mixed  data  types  and  multiple  reasoning  modes  including  reasoning  about 
time  and  uncertainty. 

The  hybrid  knowledge  base  (HKB)  that  was  proposed1  consists  of  a  set  of  statements  of  the 
form: 

A  :  [ttg,<o]  *-  S  ||  Bi  :  [iii,<i] k...LBn  :  [u„,tn] 

where  A ,  B\,  ...,£„  are  atoms  of  an  underlying  logical  language  L,  S  is  a  constraint  over  a  con¬ 
straint  domain,  E,  and  each  tij  is  a  non-negative  real  number  and  is  a  set  of  non-negative  real  num¬ 
bers.  E  is  railed  the  constraint  part  of  the  above  clause,  and  A  :  [tto,  to]  *-  B\  :  [t*i,  &...&£„  : 

[tin,  t„]  is  called  the  annotated  clause  part  of  the  above  formula.  The  intuitive  reading  of  the  above 
clause  is:  “If  E  is  true  with  respect  to  the  constraint  domain  E,  and  for  all  1  <  t  <  n,  Bi  is  true 
with  a  certainty  of  tq  or  more  at  all  time  points  in  V  then  A  is  true  with  certainty  uo  or  more  at 
all  time  points  in  to-” 

Nerode  et  at.1  extend  HKB  theory  to  include  non-monotonic  modes  of  negation  as  well,  but  for 
the  purposes  of  this  paper,  we  will  consider  only  positive  HKBs  obtained  as  follows:  first,  E  is  the 
domain  of  real  numbers  R*.  Thus,  all  constraints  occurring  in  HKB  clauses  are  over  the  real 
domain.  Second,  we  assume  that  time  is  static,  and  uncertainties  are  not  present.  This  means  that 
the  t,’s  all  refer  to  the  same  point  in  time,  and  the  tit’s  are  all  1. 

Development  of  efficient  implementation  paradigms  for  the  full  range  of  features  encompassed 
by  the  HKB  framework  is  a  challenging  problem.  A  mixed  integer  linear  programming  methodology 
has  been  developed  by  Jeroslow2  and  by  Nerode  and  Subrahmanian3  to  compute  stable  models  for 
function-free  Prolog-like  logic  programs.  In  order  to  handle  nonlinear  constraints  (without  reducing 
them  to  linear  constraints),  a  different  methodology  must  be  adopted. 

In  the  class  of  Constraint  Logic  Programming  (CLP)  languages,  due  to  Jaffar  and  Lasses4,  linear 
and  nonlinear  constraints  in  extended  logic  programs  are  solved  by  a  combination  of  techniques 
that  includes  linear  programming  methods  and  unification.  CLP(R),  a  constraint  programming 
language  over  the  domain  of  real  numbers,  solves  linear  constraints  by  generalizing  unification  and 
solves  nonlinear  constraints  by  delaying  the  solution  until  a  sufficient  number  of  variables  have  been 
solved  to  reduce  the  nonlinear  constraints  to  linear  constraints.  CLP(M),  a  constraint  programming 
language  for  solving  optimization  problems,  uses  the  "ConstrainedMin”  and  ”  ConstrainedM ax” 
routines  in  Mathematica  to  solve  embedded  optimization  problems. 

In  this  paper  we  describe  a  neural  network  approach  for  solving  logical  deployment  problems 
expressed  within  the  hybrid  knowledge  base  framework.  The  approach  concentrates  on  solving  lope 
programs  which  contain  clauses  requiring  satisfaction  of  geometric  constraints  and  optimization 
problems  in  conjunction  with  logical  constraints.  The  constraints  express  facts  and  relationships 
describing  terrain  characteristics,  terrain  reasoning,  and  user  requirements. 

The  remainder  of  the  paper  is  organized  as  follows.  Section  2  introduces  the  nonlinear  op¬ 
timization  procedure  and  an  illustrative  site  identification  system.  Section  3  discusses  the  neural 


network  algorithm  and  numerical  simulator.  Section  4  contains  experimental  results  obtained  using 
the  neural  network  approach  on  the  site  identification  system.  Conclusions  and  recommendations 
for  future  research  are  briefly  outlined  in  Section  5. 

2.  NONLINEAR  OPTIMIZATION  PROCEDURE 

The  goal  of  the  nonlinear  optimisation  procedure  is  to  map  deductive  systems  with  embedded 
geometric  constraints  and/or  embedded  optimisation  problems  into  combinatorial  optimisation 
problems.  Combinatorial  optimisation  problems  can  be  defined  as  the  class  of  problems  which  deal 
with  maximising  or  minimising  a  function  of  many  variables  subject  to  inequality  and  equality 
constraints  and  integrality  restrictions  on  some  or  all  of  the  variables.  We  will  be  concerned  with 
the  equality-constrained  optimisation  problem 

minimise  /(*), 

subject  to  h(x)  =  0,  (1) 

where  /  :  Rn  — »  R,  h  :  Rn  — >  Rm  and  /  and  h  are  nonlinear  functions. 

The  procedure  to  solve  a  logic  program  with  recurrent  neural  networks  consists  of  the  following 
three  steps: 

1.  Construction  of  Completion(Ground(P)), 

2.  Transformation  of  Completion(Ground(P))  into  a  Combinatorial  Optimisation  Problem,  and 

3.  Transformation  of  the  Optimisation  Problem  into  an  Energy  Minimisation  Problem. 

2.1.  Construction  of  completion(ground(P)) 

In  order  to  construct  the  completion  of  the  ground  version  of  a  logic  program  P,  variable-free 
clauses  in  ground(P)  are  converted  into  equivalence  clauses.  The  ground  version  of  P  is  obtained 
by  replacing  any  logic  clause  in  P  that  contains  variables  with  the  set  of  all  clauses  obtained  by 
substitution  of  atoms  for  variables.  In  the  completion  of  the  ground  version,  no  two  clauses  have 
the  same  head  and  the  implication  operator  is  replaced  by  the  equivalence  operator.  For  example, 
if  there  are  two  or  more  clauses  with  the  same  head,  they  can  be  combined  as  below: 

Bi,i  &  Bi'2  &  ...  &  Bilfl  — >  A 
Bk,  1  &  -®Jk,2  &  ...  &  Bk,n  *  A 

can  be  rewritten  as 

(Bi,i  &  Ri,j  &  ...  &  Bitn)  V  ...  V  ( Bk,i  &  Bk, 3  &  ...  &  Bk,n)  *-*  A 

A  more  detailed  description  of  the  procedure  for  construction  of  the  completion  of  the  ground 
version  of  a  logic  program  can  be  found  in  Bell  et  al .3 


For  the  site  identification  system  in  Figure  1  below,  the  results  of  the  first  step  of  the  pro¬ 
cedure  for  the  rules  bestJiovse(X,Y),  existing. construdion(Xt  Y),  possible  building  Mte(X,Y), 
and  possiblejrestaurant.site(X,  Y)  are  respectively 

bestJumse(at  b)  *-*  house(a,  b)  Sc 

(a,  b)  =  arg{max  ( distance  from  (R,  S)  to  (2, 2))}. 
existing. construction(a,  6)  *->  chemical ,plani(a,b)  V  house(a,b )  V  inter  state(a,b). 

possible  Jbuilding^site(a,  b)  «-»  flatJerrain(a ,  b)  Sc  ->  existing ,construction(a,  b). 

possible  jrestaurant.site(a,  b)  *-+  possible  Jyuildingsite(a,  b)Sca>  3Scb  >  3Sc 

(distance  from  (a,  b)  to  (5, 5)  <  1.2). 

where  a  and  b  are  constants  obtained  by  substituting  one  of  the  values  1,2, 3, 4, 5  for  X  and  Y . 


Logic  Program  P: 
chemical  jUant  (2, 2). 


flatjterrain(l,  1). 
flatJLerrain{2t  3). 
flatJerrain(4, 1). 
flatJerrain(5, 3). 


flat.terrain(  1,2). 
flat.terrain(2t 4). 
flat.terrain(4, 2). 
flat.terrain(5, 4). 


flatJerrain(l,3). 
flatJterrain(3, 1). 
flatJerrain(4, 4). 
flatJerrain(  5, 5). 


flatJerrain(2 , 1). 
flatJterrain(3, 2). 
flatJLerrain(  4, 5). 


flat.terrain(2, 2). 
flat.terrain(3, 3). 
flatJerrain(  5, 2). 


house(l,4).  house(l,5).  house(  2,5). 


interstate^  5, 5). 


lake(  5, 1). 

existing. construction(X,Y)  chemical  ,plant(X,Y). 

existing. construction(X,  Y)  «-  housc(X,  Y ). 
existing. construction(X,  Y )  «—  inter  state(X,Y). 


possible  Jbuilding^site(X,  Y)  «—  flatJerrain(X,  Y)  Sc  existing. construction(Xt  Y  ). 

TYPE  I  EXTENSION  (Embedded  Geometric  Constraint): 

possible.rcstaurant^ite(X, Y)  «-  possible  Jmildingjsite(X,Y)  Sc  X  >  3ScY  >  3b 

( distance  from  (X,  Y)  to  (5, 5)  <  1.2). 

TYPE  II  EXTENSION  (Embedded  Optimization  Problem): 

bestJumse(X,Y)  <-  house(X,Y)  Sc 

( X,Y )  =  arg{max  (distance  from (Rt S)  to (2, 2),  £,5e{l,2,3,4,5})}. 


Figure  1:  Site  Identification  System 


2.2.  Construction  of  the  combinatorial  optimisation  problem 

The  next  step  in  the  nonlinear  optimisation  procedure  is  to  construct  the  constraint  func¬ 
tions  and  the  objective  function  for  the  combinatorial  optimisation  problem.  Each  logical  predicate 
appearing  in  the  completion  of  the  ground  version  of  a  logic  program  is  associated  with  a  real  vari¬ 
able  in  the  range  [0, 1]  with  0  denoting  a  true  predicate  and  1  a  false  predicate.  Upper  and  lower 
boounds  are  established  for  any  slack  variables  which  may  be  added  when  transforming  inequality 
constraints  in  the  logic  program  into  equality  constraints  for  the  combinatorial  optimisation  prob¬ 
lem.  For  logic  programs  with  no  extensions,  the  objective  function  is  formulated  as  the  negative  of 
the  sum  of  the  associated  variables.  Minimising  the  objective  function  while  satisfying  the  system 
of  constraints  given  in  equation  (1)  ensures  that  any  logical  predicate  which  cannot  explicitly  be 
inferred  as  true  will  receive  a  value  of  false. 

The  system  h(x)  =  0  of  numerical  constraints  represents  the  following  categories  of  logical 
constraints: 

•  Fact  Constraints 

•  Domain  Constraints 

•  Logic  Constraints 

•  Geometric  Constraints 

•  Optimization  Constraints 

Fact  constraints  are  represented  by  equating  the  associated  variable  to  zero.  For  Logic  Program 
P,  there  are  24  facts  in  the  completion  of  the  program,  each  of  which  has  a  corresponding  variable 
associated  to  it.  The  fact  chemical jplant{2, 2),  for  example,  is  represented  by  a  variable  £3,3. 
The  constraint  equation  that  requires  this  fact  to  be  true  is  (73,3  =  0.  £3,3  will  be  zero  when  the 
corresponding  predicate  is  true  and  non-zero  when  the  corresponding  predicate  is  false. 

Domain  constraints  are  also  imposed  on  the  values  of  the  associated  variables.  In  order  to 
ensure  that  the  value  of  each  associated  variable  will  be  either  true  or  false ,  constraints  of  the  form 
V(1  -  V)  =  0  must  be  included  for  each  variable  associated  with  a  logical  predicate.  For  the  site 
identification  system  without  any  extensions,  there  are  a  total  of  175  domain  constraints.  Each  of 
the  seven  predicates  in  the  program,  i.e.  chemical4>lant(X,Y),  flatJerrain(X,Y),  house(X,Y), 
inter  state(X,Y),  lake(X,Y),  existing -canstruction(X,  Y),  and  possible Jruildingjite(X,Y),  ex¬ 
pands  into  25  predicates  in  the  program  completion,  making  a  total  of  175  associated  variables. 

Logic  constraints  for  a  given  clause  are  represented  by  equating  to  zero  functions  that  are  non¬ 
zero  for  a  combinations  of  truth  values  that  is  logically  invalid  and  which  are  zero  for  all  logically 
valid  sets  of  truth  values.  The  construction  of  appropriate  sets  of  functions  can  be  completely 
automated.  First,  we  will  illustrate  the  technique  for  selected  logic  clauses  in  Logic  Program  P  and 
then  we  will  outline  the  general  procedure. 


The  technique  for  the  function  construction  is  illustrated  below  for  the  two  clauses  in  Logic 
Program  P,  existing  .construction^,  b)  and  possible  Jmildingjsite{a,  b).  The  following  notation  will 
be  used.  Associated  with  predicates  existing. construction(a,  b),  chemical  j>lant(a,b),  house(a,b), 
inter state(a ,  6),  possible Jbuilding^ite(a,  b),  and  flat. ter  rain(a,  b)  are  the  variables  Eajb,  Ca,b,  Hajb, 
Ia,bt  Pa,b}  and  Fa<b,  respectively.  The  invalid  sets  of  truth  combinations  for  (Eajb,Caj,,Haj>Jajk)  are 
{t,f,f,f},  {f.t.t.t},  {f,t,t,f},  {f,t,f,t},  {f,t,f,f>,  {f,f,t,t>,  {f,f,t,f>,  and  {f,f,f,t}  where  t  denotes  true  and 
/denotes  false.  The  invalid  sets  of  truth  combinations  for  (Pa^,Fajb,Ea^)  are  {t,t,t},  {t/,t},  {t,f,f}, 
{f,t,f}.  For  each  invalid  truth  combination,  a  function  is  formed  by  associating  a  value  of  (1  —  V) 
to  a  truth  value  of  t,  a  value  of  V  to  a  truth  value  of  /,  and  then  multiplying  together  the  terms 
corresponding  to  truth  values  appearing  in  an  invalid  combination. 

The  corresponding  constraints  for  existing  .construction^,  b)  and  for  possible  Jruildingjsite(a,  b) 
are  given  below: 


Logic  Constraints  for  existing_construction(a,b) 

(1  —  Ea,b)Ca,bPa,bIa,b  =  0 

£«,fc(i  -  c«,*)(i  -  ira,k)(i  -  Ia,b)  =  0 

Ea,b(,  1  -  Cait)(l  -  Hajb)Ia,b  =  0 

Ea,b(l-Ca,b)Ha,b(l-IaJb)  =  0 

1  —  Cttlb)Ha,bIa,b  =  0 
Ea,bGa,b{  1  —  Bafi)(l  —  Ia,b)  =  0 

Fa,bCa,b{  1  "  Ba,b)l a, 6  =  0 

EaJb^aJbP a,i(  1  "  fo,l)  =  0 

Logic  Constraints  for  possible_building_site(a,b) 

(  -Pa,6)(l-Fa,6)(l-Ea,k)  =  0 

(1  -  Pa,b)Fajb(l  -  EaJb)  =  0 

(1  —  Pa,b)Pa,bEaJb  ~  0 

•Po,t(l  ~  Fa,b)Ea,b  =  0 


The  general  procedure  for  construction  of  functions  for  nonlinear  equality  constraints  to  repre¬ 
sent  logic  constraints  consists  of  the  following  steps: 

1.  Enumerating  Invalid  Truth  Value  Combinations  for  Clauses  in  Completion(Ground(P)), 

2.  Associating  true  to  a  term  of  the  form  (1  -  V)  and  false  to  a  term  of  the  form  V,  and 

3.  Multiplying  Associated  Terms  for  Each  Invalid  Combination. 


Numeric  and  geometric  constraints  embedded  in  rules  are  treated  analogously  to  logical  con¬ 
straints.  During  the  process  of  substitution  of  values  for  variables  in  the  first  stage  of  the  procedure, 


several  simplifications  may  be  directly  applied.  For  example,  in  the  site  identification  system,  pos¬ 
sible  sites  considered  for  a  restaurant  may  be  limited  to  those  sites  where  X  and  Y  are  both  greater 
than  3.  This  means  that  there  are  possibly  four  sites,  sites  (4, 4),  (4, 5),  (5, 4),  and  (5, 5),  which  need 
to  be  considered  in  the  formulation  of  the  optimisation  problem.  All  possible  truth  combinations 
for  the  rule  for  possible.restaurant.site  are  then  considered.  The  inequality  constraint  that  the 
distance  between  a  possible  restaurant  site  and  the  site  (5, 5),  where  the  interstate  is  located,  must 
be  less  than  1.2  is  turned  into  an  equality  constraint  by  adding  a  slack  variable.  Thus,  the  distance 
condition  inequality  d((a,6),( 5,5))  <  1.2  becomes  a  distance  condition  equality  of  the  form 

d((a,  b),  (5, 5))  -  1.2  +  slack(a,  b)  =  0  (2) 

where  slack(a,b)  is  the  slack  variable  associated  with  d((o,6),(5,5)).  Let  us  denote  the  square  of 
the  expression  on  the  left-hand  side  of  the  equality  in  equation  2  above  as  Dist*  h.  Whenever,  in 
one  of  the  invalid  sets  of  truth  combinations,  the  truth  value  for  the  distance  condition  evaluate 
to  false ,  Disti  l  is  inserted  into  the  associated  optimisation  constraint  product.  When  the  truth 
value  for  the  distance  condition  is  true,  a  desirable  function  for  the  associated  term  in  the  product 
would  have  the  property  of  being  close  to  zero  when  the  distance  condition  is  not  satisfied  and 
non-zero  if  the  distance  condition  is  satisfied.  A  function  which  we  have  used  in  our  experiments 
is  1/(1  +  exp(Distl  b)).  Using  this  function,  the  logic  constraints  for  possiblejrestaurant.site  are 

Geometric  Constraints  for  possible_restaurant_site(a,b) 

(1  -  Ratb)Pa,b  =  0 

(l-RaJb)(l-PaJb)Distljb  =  0 

RaA  1  -  P-.sKl/a  +  expiDistlj,)))  =  0 

where  Ra,b  and  Pa,b  are  the  variables  associated  respectively  with  possible jrestaurantsite  and 
possible Jbuilding^ite(a,  6),  respectively. 

Embedded  optimization  problems  in  logic  clauses  effect  both  the  format  of  the  optimization 
constraints  and  the  format  of  the  objective  function.  If  the  sample  logic  program  consists  only 
of  logical  information  or  logical  information  combined  with  geometric  constraints,  the  objective 
function  to  be  minimized  is  simply  the  negative  of  the  sum  of  the  associated  variables.  With  an 
embedded  optimization  problem,  both  the  negative  of  the  sum  of  the  associated  variables  and  the 
embedded  optimization  condition  must  be  simultaneously  optimized.  In  addition  to  the  logical 
constraints,  a  uniqueness  constraint  must  be  added  to  ensure  that  exactly  one  solution  is  found  to 
the  optimization  problem.  The  form  of  the  constraints  for  the  bestJwuse  clause,  which  contains 
an  embedded  maximization  problem,  is  illustrated  below: 

Optimization  Constraints  for  best_house(a,b) 

24  -  £  Ba,b  =  0 

atk«{l,al3,4,S} 

(1  ~  Ba,b)HaJb  =  0 

where  Ba,b  and  Ha,b  are  the  variables  associated  respectively  with  bestJumse(a,b)  and  house(a,b). 
The  form  of  the  objective  function  for  the  Site  Identification  System  consisting  of  Logic  Program 
P  with  TYPE  II  extension  for  bestJwuse{a,  b)  is 


Objective  Function  for  Logic  Program  P  with  TYPE  II  Extension 

—  ^  CaJk  +  +  £|>  +  PmJS  +  ^t> 

•,S«{1,2,3,4,5} 

^2  ( distance  from  (a,  b)  to  (2, 2))  (1  -  Bmjk) 

«>€{l,a, 3, 4, 8} 

where  Bajb,  Caj>,  Ea*t  F.j>,  Hajk,  hjk,  L«>  and  PaJk  are  the  variables  associated  respectively  with 
bestJwuse(a,b)t  chemical  4>Iant(a,b),  existing  .construction^,  b),  flatJerrain(a,b),  house(a,b), 
inter state(a,b),  lake(a,b),  and  possible  JmUding^site(a,  b). 

2.S.  Transformation  of  optimization  problem  into  energy  minimization  problem 

The  goal  of  this  step  of  the  nonlinear  optimisation  procedure  is  to  transform  the  combinatorial 
optimisation  problem  into  an  energy  minimisation  problem  which  can  be  solved  by  recurrent  neural 
networks.  An  energy  function  is  constructed  consisting  of  the  sum  of  the  constraint  functions  and 
the  objective  function.  The  basic  form  of  the  resultant  energy  function  E  is 

E  =  53  ^("violation  of  constraint  i")  +  B(ncost”)  (3) 

i 

where  Ai,  B  >  0  and  "cost”  is  an  optimisation  cost  function  that  is  independent  of  constraint 
violations.5 

3.  NEURAL  NETWORK  MODEL 

The  motivation  behind  using  neural  network  approaches  for  solving  combinatorial  optimisation 
problems  is  to  find  near-optimal  solutions  in  reasonable  amounts  of  time  to  problems  of  realistic 
sizes.5  Many  combinatorial  optimisation  problems,  such  as  the  Traveling  Salesman  Problem,  are 
NP-complete  problems  (problems  whose  solution  time  is  of  order  e*p( »)).  For  this  class  of  problems, 
which,  in  addition  to  being  NP-complete,  typically  have  associated  energy  functions  with  multiple 
local  minima,  Looi5  points  out  that  "optimal  solutions  are  unattainable  for  problems  of  realistic 
sizes”  and  that  "neural  network  models  can  be  promising  for  certain  optimization  problems  which 
can  be  readily  expressed  as  the  minimization  of  an  energy  function”. 

Recurrent  neural  networks  are  single-layer  feedback  networks  which  represent  nonlinear  dynam¬ 
ical  systems.  The  state  trajectories  of  the  networks  evolve  over  time  to  attractors  which  tend  to 
minimize  an  associated  energy  function.  Specific  classes  of  recurrent  neural  networks  have  been 
identified  which  are  asymptotically  stable  and  which  can  generate  optimal  solutions  to  linear  pro¬ 
gramming  problems.  Hopfield6  and  Cohen  and  Grossberg7  have  shown  that  discrete  and  analog 
recurrent  neural  networks  with  no  auto-connections  and  with  symmetric  connection  weights  are 
asymptotically  stable.  Wang5  has  developed  sufficient  conditions  for  asymptotic  stability  of  analog 
recurrent  neural  networks  with  time-varying  interconnection  weights.  Wang5  also  developed  suf¬ 
ficient  conditions  for  the  feasibility  and  optimality  of  solutions  to  convex  programming  problems 
generated  by  this  class  of  recurrent  neural  networks. 


A  numerical  simulator  for  recurrent  neural  networks  with  time-varying  penalty  parameters, 
which  was  developed  by  Wang8  and  modified  based  on  suggestions  by  the  author,  was  used  to  solve 
the  energy  minimisation  problems  obtained  by  applying  our  nonlinear  optimisation  procedure  to 
logic  programs.  A  brief  overview  of  Wang’s  technique  (as  used  in  our  experiments)  appears  below. 
Experimental  results  for  the  Site  Identification  System  (Figure  1)  are  described  in  the  next  section. 


S.l.Energy  Function  Design 

The  first  task  in  the  development  of  a  recurrent  neural  network  for  solving  an  optimisation 
problem  is  to  design  an  energy  function  whose  minimum  represents  the  solution  to  the  optimisa¬ 
tion  problem.  Let  us  assume  that  the  optimisation  problem  can  be  formulated  as  a  nonlinear 
programming  problem  with  a  single  objective  function  /(if)  in  which  the  goal  is  to 

minimise  /(if)  subject  to  the  constraint  p( if)  =  0 

The  penalty  function  p(if)  is  assumed  to  be  a  non-negative,  differentiable  function  which  is  equal 
to  zero  if  and  only  if  v  is  a  feasible  solution  to  the  optimization  problem.  An  energy  function  can 
then  be  defined  as 

The  penalty  parameter  A(f)  is  assumed  to  be  a  positive,  monotonically  increasing  function  of  t 
and  v(t)  which  is  initialized  to  a  low  value  and  increased  slowly  to  avoid  numerical  instability  and 
overpenalization.  A  suitable  function  for  A(t)  is 


S.2.  Algorithm  for  numerical  simulation 

The  next  task  is  to  develop  an  update  algorithm  so  that  the  system  will  evolve  in  the  general 
direction  of  the  negative  gradient  of  the  energy  function9  to  a  stable  minimum.  For  the  numerical 
simulation,  the  network  was  discretized  based  on  the  first-order  Euler’s  method.  In  addition  to 
selecting  a  function  for  the  penalty  parameter,  the  capacitive  parameter  must  be  determined 
and  activation  functions  F  must  be  selected  for  each  neuron.  Sigmoidal  activation  functions  are 
typically  used  for  mapping  outputs  back  to  inputs  in  recurrent  neural  networks.  For  a  variable 
ranging  between  [0,  Af],  a  sigmoidal  activation  function  may  be  constructed  as  follows: 

M 

vi[t)  =  — — —  where  ttj.Vj  are  the  input,  output  respectively  for  neuron  i 

1  T  * 

The  following  update  algorithm  (slightly  modified  from  the  algorithm  in  Wang3)  was  used  for 
the  experiments: 

Step  0:  Input  values  for  the  parameters  At,  c\,  A(0),  and  e  (used  in  the  termination  criterion). 
Also  input  initial  values  for  the  vector  v  and  determine  the  corresponding  initial  input  vector 
u  by  taking  the  inverses  of  the  sigmoidal  activation  functions. 


Step  1:  Evaluate  the  gradient  by  computing 

#W  =  vt/(«(0)  +  a(0vti>(«W) 

Step  2:  Compute  the  new  state: 

5(t  +  At)  =  fl(t)  -  A tflt)p(*)cM 
9{t  +  At)  =  +  At] 

The  capacitive  parameter  is  defined  as  and  F  denotes  sigmoidal  transformations.  The 
parameter  should  be  selected  sufficiently  small  to  make  the  network  stabilise  quickly. 

Step  S:  Adapt  the  penalty  parameter: 

if  E(t)  <  E(t  +  At)  +  e,  then  A(t  +  At)  =  A(t)  + 

Vly)ex 

Step  4:  Check  for  termination: 

if  p(i?)  >  e  then  STOP  else  go  to  Step  1. 


4.  EXPERIMENTAL  RESULTS 

Experiments  were  conducted  using  the  neural  network  approach  on  the  Site  Identification 
System  for  (1)  Logic  Program  P,  (2)  Logic  Program  P  with  the  TYPE  I  extension,  and  (3)  Logic 
Program  P  with  the  TYPE  II  extension.  The  initial  terrain  configuration  specified  by  the  set  of 
facts  in  Logic  Program  P  is  illustrated  in  Figure  2  below.  All  variables  associated  with  known  facts 
or  with  predicates  which  did  not  appear  at  the  head  of  non-empty  clauses  were  pre-set  to  0  or  1 
respectively  and  not  allowed  to  change  during  the  neural  network  update  process. 

For  the  first  case,  Logic  Program  P,  convergence  to  a  correct  solution  was  obtained  alter  only 
three  iterations.  The  twenty-five  variables  associated  with  the  existing  .construction  clause  and 
the  twenty-five  variables  associated  with  the  possibleJmildingsite  clause  were  correctly  identified 
as  0  for  true  and  1  for  false.  The  results  are  illustrated  in  Figure  3  below. 

For  the  second  optimization  problem,  Logic  Program  P  with  TYPE  I  extension,  convergence 
was  obtained  in  under  10,000  iterations  to  the  correct  solutions.  Figure  4  shows  the  solutions 
obtained  for  the  four  variables  associated  with  possiblejrestawantsite  and  for  the  fifty  variables 
associated  with  the  possible Jruilding site  and  existing. construction  clauses.  The  neural  network 
correctly  deduced  that  a  restaurant  could  not  be  built  on  site  (5,5)  because  it  was  not  a  possible 
building  site  and  that  a  restaurant  could  not  be  built  on  site  (4,4)  because  it  was  not  within  a 
distance  of  1.2  from  the  interstate  site.  Sites  (4,5)  and  (5,4)  were  found  to  be  possible  sites  for 
building  a  restaurant. 
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Figure  2:  Initial  Terrain  Configuration  for  Site  Identification  System:  (a-e)  sites  for  chemical  plant, 
flat  terrain,  house,  interstate,  and  lake,  respectively 
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Figure  3:  Solutions  to  Nonlinear  Optimization  Problem  for  Logic  Program  P:  (a-e)  sites  for  chemical 
plant,  flat  terrain,  house,  interstate,  and  lake,  respectively;  (f)  solutions  for  existing  construction 
sites;  (g)  solutions  for  possible  building  sites 


Figure  4:  Solutions  to  Nonlinear  Optimization  Pfoblem  for  Logic  Program  P  with  TYPE  I  Ex¬ 
tension:  (a-e)  sites  for  chemical  plant,  flat  terrain,  house,  interstate,  and  lake,  respectively;  (f) 
solutions  for  gristing  construction  sites;  (g)  solutions  for  possible  building  sites;  (h)  solutions  for 
possible  restaurant  sites 
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Figure  5:  Solutions  to  Nonlinear  Optimization  Problems  for  Logic  Program  P  with  TYPE  II 
Extension:  (a-e)  sites  for  chemical  plant,  flat  terrain,  house,  interstate,  and  lake,  respectively;  (f) 
solutions  for  existing  construction  sites;  (g)  solutions  for  possible  building  sites;  (h)  solutions  for 
best  house  site 


In  the  experiments  conducted  on  the  network  for  the  third  optimization  problem,  Logic  Program 
P  with  TYPE  II  extension,  convergence  was  again  obtained  in  under  10,000  iterations.  The  best 
house,  i.e.  the  one  furthest  away  from  the  chemical  plant,  was  found  to  be  the  house  at  site  (5,1). 
The  solutions  for  this  problem  are  illustrated  in  Figure  5  above. 

In  our  experiments,  parameters  and  initial  values  had  to  be  set  appropriately  to  prevent  in¬ 
stability  (oscillation  or  lack  of  convergence)  and  convergence  to  local  minima.  All  variables  in  our 
experiments  that  were  not  associated  with  facts  or  with  predicates  that  did  not  appear  at  the  head 
of  non-empty  clauses  were  initialized  to  0.5.  Typical  values  for  A(0)  were  0.01,  for  At  were  0.00001, 
and  for  c\  were  0.001.  The  network  first  gravitated  toward  the  unconstrained  solution  and  then, 
as  a  result  of  the  time-varying  penalty  parameter,  converged  toward  a  feasible  solution. 


5.  CONCLUSIONS 

This  research  has  provided  a  design  methodology  for  executing  queries  to  hybrid  knowledge 
bases  with  extended  logic  programs  containing  embedded  geometric  constraints  and  embedded 
optimization  problems.  A  particularly  innovative  aspect  of  our  approach  is  the  encoding  of  logical 
information  with  geometric  constraints  as  an  energy  minimization  problem  that  can  be  solved  with 
recurrent  neural  networks.  The  feasibility  of  the  approach  has  been  demonstrated  on  a  small,  but 
realistic,  example.  The  conclusion  reached  is  that  this  method  has  the  potential  to  compute  fast 
and  efficient  solutions  to  logical  deployment  problems  and  that  it  should  be  the  subject  of  more 
investigation  in  order  to  realize  its  full  potential  in  real-world  applications. 

Future  research  directions  include: 

•  Investigation  of  Synergistic  Algorithms. 

Synergistic  algorithms  that  combine  the  strengths  of  traditional  mathematical  programming 
techniques,  neural  network  models,  and  genetic  algorithms  need  to  be  investigated  in  order  to 
build  fast,  robust,  and  efficient  systems  for  solving  logical  deployment  problems.  Traditional 
mathematical  programming  techniques  provide  a  well-understood  foundation  for  solving,  in 
particular,  linear  and  mixed  integer  programming  problems.  Neural  networks  provide  a  fast 
and  efficient  architecture  for  local  search  and  for  computing  feasible  solutions  to  nonlinear 
constraint  problems.  Genetic  algorithms  perform  efficient  global  search  but,  when  the  objec¬ 
tive  function  is  a  time- varying  function  which  asymptotically  resembles  the  penalty  function, 
the  mating  of  two  distinct  feasible  solutions  may  not  result  in  a  feasible  solution13. 

•  Extension  to  Full-fledged  Hybrid  Knowledge  Bases. 

Methods  need  to  be  developed  to  process  queries  to  logic  programs  which  handle  classical  and 
non-monotonic  negation,  reasoning  about  time,  and  reasoning  about  uncertainty.  The  results 
of  our  present  study  and  the  success  of  neural  network  methodology  in  handling  reasoning 
about  time  and  reasoning  about  uncertainty  augur  well  for  future  investigations  in  neural 
network  methodology  for  implementation  paradigms  for  full-fledged  hybrid  knowledge  bases. 
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